Business-to-business (B2B) electronic commerce, until recently, was conducted using proprietary protocols and/or a few products at a particular standard layer (such as, for example, the International Standard Organization (ISO) Open System Interconnect (OSI) session, presentation and application layers). With the advent of new protocols for B2B data transmittal such as objects, Applicability Statement 1 protocol (AS1), Applicability Statement 2 protocol (AS2), Requests for Comments (RFC), and Electronic Business using eXtensible Markup Language (ebXML) messaging, along with business process document routing through computer systems and different applications, registries, repositories, XML based business documents, and application-to-application mapping software, products based on standards are becoming available, which while often conformant to a standards specification, are not know to be interoperable which is a must in B2B electronic commerce.
Certain current automated systems perform conformance testing but do not perform interoperability testing in a manner which supports the needs of the business community. The systems that do perform automated interoperability testing are limited in that they are not built to work In Situ, in the actual environments they are to be used in. Therefore, such automated interoperability validation and testing is not usable in a live supply chain, value chain, or distribution channel environment.
Other systems that are currently available for performing interoperability tests various problems that are not solvable without introducing an automated interoperability test system of the present invention. Such systems perform interoperability testing, for example, in certain phases such as a debug phase, a dry-run phase, and a final phase. The debug phase often conducts hundreds of individual tests against other products. The tests are designed to identify bugs, are very incremental in nature, and generally take a few months to conduct (with a set of software vendors, for example). Such vendors typically record by hand messages received and sent by all other vendors in the group, and problems are typically discussed on daily conference calls.
The dry-run phase typically consists of nearly 10% of the number of actual tests as compared to the debug phase. Preliminary tests are assumed to have successfully identified and corrected bugs, hence incremental analysis of the components needs be performed. These tests generally take around one week to conduct. The final phase, which demonstrates interoperability, predominantly repeats the steps of the dry-run phase. This ensures that a setup issue, which may have been encountered during the dry-run phase, does not preclude a product from passing the interoperability test. The complexity of this phase is similar as that of the dry run phase. This test generally take around one week to conduct.
There are a number of problems and limitations associated with the previous phases. For example, as products enter production, software bug fixes and minor enhancements are typically made. These changes most often render the product un-interoperable, despite the long effort in testing. Further, testing currently takes too long and manpower test costs are too high for a software vendor to continually support. These problems and limitations require automated interoperability test systems, specific to individual supply chains, to ensure those supply chains remain highly available.
Certain associations and organizations have testing labs for some hardware and basic software standards. However, the systems in these labs do not generalize well and often require significant setup and concurrent location of all products at the lab for testing. While these current systems and methods in testing may work well for specific standards or a set of closely related standards, they do not generalize well to a general test process. Further, the current methods do not work for automated In Situ testing within a supply chain, value chain or distribution channel where a large number of loosely coupled trading partners install and use products in uncontrolled settings. Solving this problem will require the ability to conduct ad hoc, periodic interoperability tests and In Situ validation (of the final phase type) on live production supply chain products.
Additionally, while there are a considerable and growing number of system management software vendors with products aimed at performance and availability management operations such as monitoring, event management, event correlation, and root cause analysis, none of these vendors is focused on resolving issues with higher level software such as secure messaging systems or application level automated interoperability assurance testing, validation and monitoring.
It is therefore desirable for the present invention to overcome the problems and limitations described above that are involved in interoperability testing.